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How robust arc distributed systems? 


K. P. Birman 


I and wmmg thu ch»p,er in November 1988, Aatly after a w 

unleashedu] the mtemet; by exploiting network Kcurity loopholes it penetrated 
and crashed large numben of machine.' Coincidentally, newspaper, wem fill* 
retrospective analyse of the 1987 stock market crash. l 2 Both events gave rise 
to speculation concerning the rabustnes of contemporary distributed wstems 
and it is to this topic that I address myself. ^ ’ 

Before beginning, it is important to recognize that these episodes also touch on 
rather deep ethical questions. One can and should ask about the propriety of 
writing and running a program that has no constructive purpose, or even 
ting small mveston against massive institutions armed with supercomputers. 

Personally, I feel that the running a worm shows a deplorable lack of judge- 
ment, and entertain some doubts about the modem stock market Nonetheless 
these conclusions are debatable, and strongly dependent on question. of carte’ 
The proem discussion focuses on a more technical issue, namely the robustness 
of distributed computing systems - against intrusions, but also in the presense of 
evous that arise commonly in distributed settings, such as failures and overloads. 
Because these issues are basically technical, one can hope to arrive at a more or 
, technical answers to them. To the extent that these lead back to philosophi- 
cal speculations, the questions raised concern implications of more technical con- 
clusions, and hence one might hope that they will be less controvertiai than 

‘ The program w«* designed to penetrate as many machines as possible using bugs and loopholes in 
UNIX communication and mail-handling software. Although apparently intended to ui^nLvely 

^ “ tnfccti00 ’” * Programming error caused the worm to replicate much faster 
chu intended. It gained access to nearly 6000 systems during a 48-hour petwd, overloading and 
crashing a large percentage. ana 

l u \ reference ® «** d ™™ c inarket decline, that occurred during a world-wide flurry 

ot program -driven trading in October of 1987. 
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20.1. Predicting the behavior of a distributed system 


Consider the problem of predicting how a distributed system will behave while it 

1 TT, W “ b "- 1 ' “P of ^ number, of co^non ‘ 
operating asynchronously from one another and hence with incomplete and 
inaccurate view, of one-another’s state. Moreover, few distributed 
operate in a steady state: load fluctuations are common as new tasks arrive and 

atdLXS J ° mtly ’ ** “*** nute U * «"*" 

For example, feedback can arise in an automated stock trading system 
because programmed trade decisions are based on market index«d^ chlnre 

^VeL^^ecT 6111 tTad T S tradln? Prog^nis operate independent^ 
r munm * i - However, if a condition provokes sell decisions 
in large number, of programs, or exceptionally large sell orders, it can reinforce 
itself by driving those indexes down, triggering waves of sales. Such a sequence 
apparently led to the 1987 crash. Whether or not one questions the use of trad- 
ing systems in general, it seems obvious that one could question the use of trad- 
ing programs subject to such behavior. What is less obvious is that these sorts of 
behavior, are unpredictable and can arise from seemingly trivial mechanisms. 

A behavior prediction problem also arose as an issue in the 1988 worm 
incident. One way to design a worm would be to write a distributed protocol 
that maintains a replicated list of currendy infected sites, by having worm pro- 
grams communicate directly with one another and monitor one another’s status 
to detect failures. Using this approach, one could maintain a very stable popu- 
lation of womu, infecting new rites in a highly controlled manner. However/ the 
protocol would be hard to design — similar problems were discussed in Chapter, 

1 4 and 15. An easier problem is to implement such an algorithm given atomic 
group addressing and broadcast primitives, but the designer of a worm cannot 
(yet) assume that such primitives are available. 

In an ill-fated decision, the designer of the 1988 worm evidently turned 
instead to a random algorithm. Under this approach, each worm independently 
makes decisions to infect neighboring rites based on probabilistic mechanisms 
Fhe resulting worm population is influenced by factor, that include the current 
population, the rate of new infections, the death rate, and the probability of a 
successful penetration of a system. For certain values of these parameters the 
worm population might well remain stable and small. However, for <xher 
values, an unstabU solution results, whereby the worm population will die out or 
grow uncontrollably. The question is thus how to pick parameter values that 
will definitely give stable populations. Unfortunately for the designer of the 
worm, problems of this sort are often intractable, and this one almost certainly 
is. Current mathematics gives little insight into how one might pick the 
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parameters to ensure stability, or even test for stability given particular ^ 

2T2r“ 1988 wonn *“ h,i “ 

It is striking that whereas the worm provoked much discussion nf a;.^u j 

jeest pSUt r r, - b - 

pa,d “ “f b r dCT rf-w, *. 

example in T “T" fcoib * d ‘ "'“ham™, for 

ication layer There is growing interest in applying thae^ 
wide range of cndcal settings. How can one* ™Le thaTT J 

secure and immune to chaotic behavior? Lacking this knowledge ,h^TT 
”7 0th f ”cWen B , perhap, whh cjophic comcqJSc^ ° n ' 
Unoi recemly a laboratory rarity, distributed systems have become pe^aaive 
and our society has come to rely on them over a five-v«r ner^ r ^ ’ 

Sated d “ bUKd “ rapu<in *' «*>«>logy i» ilEE 


20.2. Technology and social responsibility 


I believe that the inventor, of a technology assume an obligation to overcome 
flaws in that technology, especially flaws that could exact a direct human cost 

tUmCd 'T* "t*™ ***** 

that its weaknosM hi* • VTS cnt )f aI a technology, the more important 
to ** nci P ated before ^ become stumbling blocks. To fail 

W confront this i«ue m the context of distributed computer systems invites 
phazard interconnection of machines using mechanisms capable of interacting 

D m a rr CiP ^ ^ Ladrin » *»«» to the contra^ one must antici 

pate that confidential data will be increasingly often expend to intrusions that 

SE2 TTJZ' inCreaSi f gly ° ftcn * ***** to Option, and d£t 

allures or ail sorts will be increasingly common, 

1 Jb“ t « 1 r men ;- C “ be t c " Tied ev<!n *«•>»• In many cams, sober aoalyri, 
rv^L l°° n 1 technology imply Conor be perfected to the 

dSTL^^k”^ “* * viil * b1 '' if «*»- A good example, wrongly depen- 
dent on distributed computing technologies, u launch-on-waming Shvarc for 

SStSi r n ' S Zr* C "*•“ live hcTpt 

posed because human beings cannot function rapidly enough to make launch 

decisions in respome to a surprise attack. Unfortunately, the proponents of new 

weapons technologies have often overlooked weaknesses of a technology, and the 

Unuts on the degree to which it can be perfected. Can one really build a large 

^bwed system that is sufficiently robust to entrust it to perform such a crtical 

ULdc Based on the arguments that I will advance below, I think the answer is a 

negative one. It seems to me that there is an applicable “impossibility” result- 
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every bit as serious a limitation as any theoretically provable one. AnH „wi 

TvH° a f Piy m many other “““V- To establish this, howeUr, om 
I n the ease f TObUSt * *y*eni can reasonably be expected to' be. 

veneration m0rC mature technologies, such as transportation and power 

generation, organizations exist to ensure the safety of systems that 

pessimiam kT ’ l * meaaures manciate£l ® *° me areas are astonishing in their 
pe«miam about human potenaal for error and for assuming that unlikely evenr, 
wtil not only occur, but will do so at the worst po£Ttime 

ha« ^ ^orporate the most extreme measures to minimize risk. This 

has clearly reduced the potential for disaster. Yet, incidents continue to occur 
and in many cases the ways in which they occur raise new questions about £ 
whole assumption that systems of this sort can ever be made safe. ™ 

In contrast, the engineering of even the most widely used distributed sv» m . 
has been fairly informal. If trains crash and nuclear “excursions” occur 

despite every countermeasure that designers with yean of experience have 

Sh 7 kl ° ne n ° t “J** disruptions in di£ibuted sys- 

of*' 81 *? f “i y m ^ umal attention to robustness? The most common 

iTK?® bUttd **“" *“■ through low* level standards 

as for the ISO data transport protocols. However, the problem, identified above 

hT* !l?* | api l b !*5 0n . evd> andtotile extcru that applications- level standards 
have been developed, they have been premature and overly restrictive. Clearly 

one cannot define a standard for aspects of a sy*em that are still experimental’ 
Yet, it seems equally clear that ignoring these issues only encourages the con- 
struction of complex, fragile software* 


20.3s Principles for distributed computing 

One thing that we lack is a set of guiding principles to encourage the deveiop- 
ment of sound solutions to distributed computing problems. Let me propose a 
set of such principles now. p 


Assam* responsibility. 

Those who produce distributed computing software should make every effort to 
ensure that the software is safe for its intended mode of use and that it can only 
be used in the intended way. And, we must accept our responsibility to apply 
the highest standards of ethical behavior in our individual research and to instill 
these standards in our students and colleagues. 

Intercom* ct Jot good reasons. 

SystOTS should be interconnected to achieve concrete objectives, not in the 
abstract belief that interconnection is a good thing. Systems that are incapable 
of interacting are incapable of compromising one another. 
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Support only necessary services. 

This is especially important for services implemented anonymously and ^ 

vided as executables (without source). For examole thr iqoo I Y ^ 9 
a bun in fh^ t fnitv — * ’ trie 1988 worm made h*a nf 

a ougs in the UNIX remote finger and mail handling uriliri« n™„' v . 

finger databases on other macWnes. vC 

1« mb^r *“ "* 001 U3dU1, **“ machiDa on which they run are made 
Include self-diagnosis and authentication mechanisms 

origin ZZT £,* "*■ “ ’ Upp0rttd ' luth “ d ““ *• 

lhe re P t^“^ y Xe. infc^ "“ 0n, “° -* **■» *• *«4 

Authentication is an issue beyond its security implications. It is widely 
^t^ sh^L PfOCCd T Sh ° Uid authemicate ** "guments. Large distribS 

XlTs^^rT 7 P PlC fUrthCr - Mech “*™ «• »3d by which 
whole system components can monitor themselves continuously, actively looking 

for inconsistencies and shutting themselves down if problems « dSwL 

™* ^ “k*" aithoush <*• «? b.iisjr^ « 

S*,T*.T dly ^ c “ “ft 01 be limited, for example by exXidy 

£1“* 7* T™"* pr08nUM - ™* »PP~ch ha, long been £2 

successfully in electronic circuit switching. 5 

Design for fault- to Uranct. 

1 T* ^nbuted systems am dmigned a. if failure, will not occur, or 
give undefined behavior ,n the pnaerue of fiulures. This is precisely the convene 
o the attitude needed when building joftware to survive a wide range of com- 

specially in light of the seScheclting 

FT* ^ T “ bu,W 1 robu « ^buted system, one musf 
assume that failures wtll occur. The choice is to try to sun-ive such events, or to 
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<ta«t than and .hut down before an incomutent or enoneou. action could 

wti^tvtlaonl SSL 

f °T * *^***« 

crally aamme mdependent, benign ^ 

failures involve only packet lens, duplication, unLquenced daUve^T^nh" 

m Al k n0t . m<35a?e COrru P Don > for 8«y. or protocol violations 
Although one can question whether failures are aJwavs hmior, , 

K SS3S SS. L-t a&=5£3= 

which arbitrary, correlated and even malicious behavior are aU treated « i’ 
able. Computation in this model requires such costlv conserwn, 7^1 P ^ U ' 
preclude the use of these “ " 

iWcr, the model require, that all imcracnont with the^^Sd* 
through a Byzantine agreement, which is often impractical For aamnU ;/? 
p^tem « capable of unlocking a door, the door woL h^e to teSShd « 

ODerated q i i pLcatC ; * Uch that thrce 0411 of four actuators would have to be 
operated sunultaneously to perform the tadc.’ Even in extreme settm« suS Z 

m C SpaCC * 5hutt ! e car *° hoId ’ ^ nxlundancy wsTfeh^d bfL c - 
apprLh applications can afford adopt the most pessimistic 

To summarize there seems to be little hope for building practical day-to^iav 
systems capable of tolerating severely incorrect or malicious behavior on the aJt 
^components. Yet, if benign behavior is as^ed, on^ a^o ££ 

possibility that a system will experience fisilure modes that violate assumn. 
aons, and ask what the impact will be and how damage £££££"+ 

Design for scale. 


Ju« u n IS common to ovenimplify isuea of fault-tolerance in dhtributed m- 
«m, quonora of Kile am often neglected. Contemporary dmributed 
become hopelealy dtfficult to manage when mom thanTfew doaen IS 
M mterconnected. Syncitu that will ituerwnneet hundred. or thotnamUtf 

rd^Tan,m° mP !r ely l . difl ’ er “' daign mindjH ’ “ which Kale i, 
Ife^ughf ^ rather than an aapect that can be dealt with a, an 

Avoid mechanisms that can cascade failures. 


5 In practice, triple modular redundancy is adequate for mo* applications. Nonetheless the Bran 

une approach requires that there be at least 3T- 1 rrv.l ™ Byzan- 

i ~ J 1 1 tocai partiapana in any orococoi thax will 

coieraxe up to T Eaiiures whiic u i i oinniny. P*** 
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^ ziz p tr 

cnct^^T 1 m T’T 14 ' Inth “' praocolj, a faifcd component may 
orvj? f broad<:M ' dellvena d>* corrupt it, software ttate. U suSTa 
W '° “™* ««* P">S™u that remained operadol^ 

““ to a£to7 P M * n> ’ ,uch P™ 0 ** <«=■>*« lack mechanisms to 

iual contamination problem, although mo« some do provide 
notification if an obvious error is detected. V ac 

A different kind of cascading can occur when machines are declare f a „u. 
due moverload. If to operadonal one, oy to 

risk becoming overloaded themselves. This, in turn, would 
adures. To avoid such problems one must either design substantial wess cap a 
city into a system (which is often too costly to be practical) or detect overlaid 

Avoid using "magic” mechanisms. 

When a large system is built out of large numbers of interacting components the 
superficially simple algorithm, they embody can misbehave in surpSLT ^ 
T£s po« special problems to the designer, of distributed system^ wh« it U 
often difficult to predict exactly how a mechanism will behave under real lauds 
For example, there is a strong temptation to include scheduling heuristics and 
active mechanisms » low levels of a system; my group did this in some parts 
the ISIS system for purposes of load balancing. Yet, short of accurately model- 

U n ° ^ t0 kn ° W ^ IocaJ °P* imi »*ion decisions will yield 
globally good behavior, or simply cause the system to “thrash”. Given the 

choice, a simple, well-understood mechanism is always preferable to a fancier 
but poorly understood one. 


20.4. Future directions 


The pnnciples enumerated above raise a tremendous number of questions about 
current and future distributed systems. It is interesting to examine some of the 
application areas that were covered in the text in this light. 

20.4.1. Scaling and administration of file systems. 

The major focus of recent work on distributed file systems has been on perfor- 
CT TM like Andrew and Sprite represent major advances over, say, the 

SUN NFS, because they make more effective use of network resourees and each- 
mg, where effectiveness is typically measured in terms of file transfer bandwidth 
access latency, and the number of user, the file server can support. These are 
extremely important issues. But, is it not somewhat narrow to orient file systems 
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so strongly towards performance considerations? 

, f°, r colder the problem of scaling and administering a large distri- 

^ ereaJ CUrTCnt fiIC ,yStCnu «« a ardLctur^ aLe 

tT^tT W1 C ^ nUln IirgC numbcrs of filc «rven of varying capacity 
w F T forT T CC of local disks wiU grow so large tha^ usm* 

d^em just for caching and temporary files will be unacceptably wasteful. Yet if l 

e s^tem is assembled out of rminpU servers, current systems provide little sup- 

^ailable^S^ Cnt F ^ CM 7 nblc ’ or for °Pdmizing the assignment of files to 
available sources. For example, no existing file system maintains the primary 

copy of a file on the disk local to a user's machine, migrating updates^! 

remote file server at penods of low load to permit backups from the server and 

or fault-tolerance. While there has been considerable work on file replication 

esystems to date have taken a fairly restricted approach to this whole issue 

This problem is not a purely abstract one. The Cornell Department of Com- 

^t^3foX^ e r r T Cnt l 1> Lt P t aCed A *? 0rdCr f ° r 25 workstatlona "hich are configured 
350Mbyte local disks. A decision was made to use the local disks only for 

swapping, temporary files and storage of immutable binaries, because the avail- 
able file systems otherwise require a great deal of human engineering to manage 
and the backup problem would become a major source of overhead Tlie 
administrative group was forced to do this becaure it lacked the personnel to 
support other general purpose uses of the local disks. 

In addition to making more effective use of replication, it is likely that future 
thl hTrfr* W, f nad L t ° c , l00 1 k b*" 1 at semandc information in order to optimize 

fi°i f CaCh fi C bajCd ° n lB For «ample, current 

UNIX-based file systems ignore information about file “type”, which forces dis- 
tributed implementations to guess the best file management policies to use. This 
po cy tes to a period when the UNIX file system was touted for its simplicity 
One cou!d question whether simplicity of this sort remains desirable. Most 
UNIX applications encode information about file type through standard exten- 
sions to file names, and the step from this to genuinely typed files is not a huge 
one_ Nforeover, information about file type is of great value in a distributed 
UNIX file system, since it helps in predicting typical modes of access, the likely 
uieome of a file, the importance of maintaining availability despite failures 
compression methods to use, etc. This list of attributes will surely grow with the 
widespread use multimedia systems. 

Tremendous advantages could be gained by implementing more sophisticated 
hie system architectures. An architecture is needed in which the various servers 
are knowledgeable about one another and cooperate directly to optimize file dis- 
tribution in response to patterns of access. Moreover, since this will require some 
amount of distributed state, the solution must be one which is fault-tolerant and 
gives well-defined consistency guarantees to file users. Lacking these possibilities, 
the extent to which file systems can be scaled is inevitably limited. 
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Imereating queroons of security and authendcadon ante in a transactional con- 

accM™onm?Lid U ^^ “ convendonal problema of 

concurrency control by their comnn °° *^* tcms < ^ e P en< l on the correct use of 
mechanism, must £ IShuT r MDm " er ’ concurrency control 
tamped ^ ^ ^ dn»- 

c“.' n 1 systtm compoaed ° f 1 ^p«dU 

z?zr~~ 

Vtftware package with troroactional inierfL ^ VOTlOT IUpP '>' ,n * 

Not o^y ts the transactional authentication problem difficult to solve, it is no, 
behavior ^s oari rfln' 1 " rf"" d °"T con<:urrenc l' «*>»1 requirements or 
implements lockrng and J? dSb « 

diat a caller acquire a write lock before calling write and a road lock bribTT 
'ZT _ when such lock, am no, Led 

coarser granularity was previously acquired How can this even be exores^d 
much less formally verified? m ex P rc * eJ » 

■ ■ NlBn '. “ n * ider ,he * anjri 'y prow™. Say that a transacdonai service is 

STL-g = S ^^can E tt I. :«T 

tro, and o'™* C °"- 

ri^Tctoi“ “ *(1 m0 “ '® den ' 4nd prlcticli »>unon to 

d “®" OT to ask these sorts of question, and to budd systems *« 
m^kmrohamnn. «ch a. thb, major problem, will ari* in attem^ to move 
these technologies out of the laboratory. ^ 

20.4.3. Replicadon*Based Systems 

Chapter 15 (Wsed the sort, of group addreaing mechanmns and rtoud 
broadcast mechanisms needed in systems that maintain replicated state ^Thc 
protocols used for this are complex, and bug, in them could crash large numbed 



472 

Jt-P. BIRMAN 

of machines. It would be hard to envision a failure mode in which =. ^ 

mi K ht ShUt d<>Wn ^ machine on a network, but it is fairly easy' 
o imagine how bugs in a complex protocol could have this effect ^ 

<vin^h COmpleXi T ha *J7” ml Certainly, work is needed on simpli- 

Jj"? u P™ 0001 * 1 ^ m repiicauon-based systems. Might there not be a L 
build these up from simple, verifiable mechanisms? Another issue is that to 
extent possible, these protocols should be implemented as part of the operat- 
ing system. My reasoning is that if operating sterns lack m^ort f<7theTra 
of mec^ms needed in application software, those applications either will not 
get buil^wtll be built using less than ideal methods or will be forced to individu- 

thc " UmnS mechanisms. Moreover, security and trust considera- 
ons would limit the composition of large systems out of smaller components- it 
will be too hard to agree on protocol specifications and implementation details. 

r- *** that a . sub8tantiai number of applications will need replication. 
Given this situation, it seems that one would be better off providing such services 
m a standard way. If individual application builders are asked to take on an 
effort of this scale, a large amount of duplicated effort will surely result, leading 

^ d«i^° n 1 “ robUat ’ 1C “ » maintain 


20.5. Changing the Way People Think about Distributed Computing 

™”**™p™*;** of a major corporation who expressed 
mterest in ISIS. This individual explained that within his company, known pri- 
marily for its mainframe systems, a perception had arisen that we need to 
change the way people think about distributed computing.” This comment is 

mmgumg at scveraJ levels: how do people think about distributed computing? 
Why change this? And how? * 5 

For too long, distributed computer systems have been viewed primarily in 
terms of interconnection. We have tended to think about such systems as a way 
to run a program on one machine that uses resources on another and to send 
mail to our colleagues, and have generally treated workstations as if they were 
terminals connected to a mainframe. The benefit of a using a workstation is 

often seen primarily m terms of its ability to offload computation fiom a central- 
used resource. 

Perhaps the time has come to recognize that centralized servers and tran- 
sparent distribution are not always good things. It seems clear that the real 
advantages to distributed systems come about only when one begins to treat the 
act of distribution as a positive element that can contribute to the solution of an 
application, rather than as an annoyance that should be concealed. This does 
change the requirements that one places on the communication support of the 
operating system, but the problems that arise can be solved. It is entirely possi- 
ble that the future successes of distributed computing will be in applications 
where decentralization, autonomy, fault- tolerance and cooperative behavior are 
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crmcal Whereas many of these applications are today viewed as either too 
difficult or too costly to solve, the technology for building such systems is finally 

Recall Fredrick Hayes- Roth’s comments, quoted in Chapter 14. Like Dorothy 

lu£n Tt Z° K WS hOUJC “ d SuddenJ y — ** world in color, the ££ 

££ IZ 2?®* aw ^° W u enor ™'*- The potendal to solve new prob- 

lems and take on new applications that this will enable staggers the imagination 

In accepting the challenges offered by these new applications, however one 

c^'r^Lw ’ n Z e TT ?°*- jUSt h °" TObUSt technoI °8 ic ^ solutions to problems 
«n possibly be. And in situations where analysis of the technical barren to a 

solution reveals that the available technology is not adequate to the tatic it is 
necessary to accept the reality of these limitations. Blind faith in technology can 
simply no longer be justified in the face of the increasingly long li* of technical 
£f“ chat the world has compiled. Those who create tiLe 

fh ±C ^^"“blUcy to do all that they can to ensure that 

thor products will be used in appropriate and reasonable ways. 

T^e era of distributed computing is just beginning. In writing this textbook, a 
group of us have come together to survey the field and point in some of die 
directions that it has to offer. In rereading the material that we have produced 

fieW^T 8 ^ T ** ** by ^ **"*”■ ^ been made thn^houTS 

ff nn^f aCCdCfatl f! T rf At the same time, a tremendous range 

of problems remain to be solved. In this respect, distributed computing seems to 
be special within computer science as a whole. Whereas many other £eas have 
flowered rapidly only to stagnate rapidly, new directions keep opening up in dis- 

^rf^Tw an lf reVOl ‘f? n> keCp OCCUrrm * “ ^en the most “traditional” 
^ V .. . Meanwhlle * more and more applications depend on some 

form of distribution, and these also pose new challenges. The potential applica- 
aons of distributed computing technology have hardly been tapped. Provided 
that this is done in a careful, considered manner, distributed computing could 

change die way that we deal with computer systems in a profound and 
beneficial way. 

It is unfortunate that the potential for abuse of this technology is a real as for 
any other technology that civilization has devired. Nonetheless, that potential u 
real, and wdl have increasingly important consequences. Our field must accept 
the responsibility for this technology: even as we create these new forms of com- 
puting, it falls upon us to control them through new and more demanding ethi- 
cal standards. 9 
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